Accuracy Realiability Metric

Transitioning from general-purpose data science to the eFabric™ ecosystem requires a move toward System-Level Reliability. In an "Always-On" environment, a model's worth is not judged by its performance on static test sets, but by its ability to maintain high fidelity while operating within the stringent energy and memory constraints of the Syntiant® NDP.

To bridge the gap between theoretical accuracy and real-world utility, we utilize a tiered metric system that prioritizes the operational integrity of the edge node.

The Hierarchy of Edge Reliability

While traditional accuracy provides a baseline, we evaluate the TML120's performance through metrics that reflect the physical consequences of inference at the edge:

• Precision and Recall: Rather than a single accuracy percentage, we focus on Recall (the ability to find all true events) and Precision (the ability to ensure a trigger is actually the target event).

• F1-Score: Given the "imbalanced" nature of the real world—where target events like a mechanical failure are rare—the F1-Score provides a harmonic mean that prevents a model from appearing successful simply by predicting "no event."

• Confusion Matrix Analysis: We map every inference into the four quadrants of the eFabric™ framework (TP, TN, FP, FN) to visualize where the model is struggling—whether it's "over-sensitive" to noise or "under-sensitive" to varied event signatures.

Quantization Fidelity

Because the NDP operates on 8-bit integer logic, we must account for the mathematical "drift" that occurs when a high-precision model is compressed. We measure this through

Accuracy Drop ($\Delta Acc$):

Formula : Accuracy Drop ($\Delta Acc$):

$$\Delta Acc = Acc_{float} - Acc_{quant}$$

A successful eFabric™ implementation typically targets a < 1%, ensuring that power savings do not come at the expense of system reliability.

---

Understanding False Acceptance (FAR) and False Rejection (FRR)

In Always-On Edge AI, reliability is defined by the balance between two competing error types. These metrics are the "North Star" for tuning eFabric™ hardware, as they directly impact both battery life and the user's trust in the device.

A. False Acceptance Rate (FAR)

A False Acceptance (also known as a False Positive) occurs when the model incorrectly identifies background noise, a non-target vibration, or an unrelated movement as a "match."

• The Technical Cause: The input data's spectral or temporal features are close enough to the target class to exceed the programmed Confidence Threshold.

• The System Impact: Every False Acceptance triggers the NDP’s interrupt pin (IRQ), forcing the high-power host controller to wake up from deep sleep to process a non-event.

• The Consequence: High FAR leads to "Battery Hemorrhage" and notification fatigue for the end user.

B. False Rejection Rate (FRR)

A False Rejection (also known as a False Negative) occurs when a genuine target event—such as a specific wake-word or a critical mechanical failure—happens, but the model fails to trigger.

• The Technical Cause: Environmental factors like extreme background noise (low Signal-to-Noise Ratio) or variations in the event signal (e.g., a different accent or a muffled sound) cause the confidence score to fall below the threshold.

• The System Impact: The device remains "blind" or "deaf" to the event it was designed to monitor.

• The Consequence: High FRR leads to "User Friction," where the product is perceived as broken or unresponsive.

C. The Mathematical Tug-of-War FAR and FRR exist in an inverse relationship. If you make the model more "sensitive" to catch every possible event (lowering FRR), you inevitably increase the number of false alarms (increasing FAR).

To visualize this trade-off, we use the Detection Error Trade-off (DET) Curve. The goal for an eFabric™ engineer is to find the Equal Error Rate (EER), which is the point where the two error rates are identical.

Equal Error Rate (EER):

$$EER = \{ x \mid FAR(x) = FRR(x) \}$$

False Acceptance Rate (FAR) Calculation:

$$FAR = \frac{FP}{FP + TN}$$

False Rejection Rate (FRR) Calculation:

$$FRR = \frac{FN}{FN + TP}$$

Definitions within the eFabric™ Framework:

Term	Full Name	Description in Edge AI
FP	False Positive	The "False Alarm." The NDP incorrectly triggers the host because it thought it heard the target event, but it was actually just noise.
TN	True Negative	The "Silent Success." The NDP correctly identifies that no target event is occurring and remains in a low-power state.
TP	True Positive	The "Clean Catch." The NDP correctly identifies the target event and successfully wakes the host controller.
FN	False Negative	The "Missed Event." The target event occurred, but the NDP failed to recognize it, leaving the system unresponsive.

🚀Tuning for the Use Case

"The 'correct' balance depends entirely on the application. For a Smoke Detector, you should prioritize a near-zero FRR (it must never miss a fire), even if it means a slightly higher FAR. Conversely, for a Smart Watch Wake-Word, you might prioritize a lower FAR to protect the small battery, even if the user occasionally has to repeat the command."

---

Confusion Matrices: Interpreting Model Results

While statistical summaries like "90% Accuracy" are useful for high-level reporting, they mask the specific nuances of model failure. The Confusion Matrix is the primary diagnostic tool used within the eFabric™ ecosystem to decode where and why a model is struggling. It provides a visual and mathematical breakdown of the four decision states (TP, FP, TN, FN) we defined previously.

A. The Diagnostic Value of the Matrix

By mapping "Actual" classes against "Predicted" classes, the Confusion Matrix allows an engineer to identify specific inter-class confusion.

• The Symmetrical Goal: In a perfect model, all values would lie on the Main Diagonal (Top-Left to Bottom-Right), representing True Positives and True Negatives.

• The Error Analysis: Values in the "Off-Diagonal" cells highlight specific weaknesses. For example, if you are training an NDP to detect "Screaming," and the matrix shows a high number of False Positives for "Whistling," you know exactly which sounds to include in your Negative Dataset (Section 4.0) to refine the model's boundaries.

B. Derivative Metrics for Edge AI

From the raw counts in the Confusion Matrix, we derive two critical percentages that define the model's "intelligence" quality:

1. Precision (Quality over Quantity)

Precision answers the question: "Of all the times the device woke up, how often was it correct?"

• Focus: Minimizing False Positives (FP).

• Application: High-precision models are essential for devices where every wake-up is costly (e.g., a satellite-linked remote sensor).

$$\text{Precision} = \frac{TP}{TP + FP}$$

2. Recall / Sensitivity (Coverage) Recall answers the question: "Of all the real-world events that happened, how many did we successfully catch?"

• Focus: Minimizing False Negatives (FN).

• Application: High-recall models are critical for safety applications (e.g., a heart-rate anomaly detector) where missing a single event is catastrophic.

$$\text{Recall} = \frac{TP}{TP + FN}$$

C. Balancing the Two: The F1-Score Because it is easy to "cheat" and get 100% Recall by simply triggering on everything, or 100% Precision by never triggering at all, we use the F1-Score. This is the Harmonic Mean of Precision and Recall, providing a single number that penalizes extreme imbalances.

$$F1 = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$$

🚀The Confusion Audit

"Always look at your Confusion Matrix before you change your model architecture. Often, the issue isn't the 'Brain' (the neural network), but the 'Fuel' (the data). If the matrix shows the NDP is confused between two similar signals, it's a sign you need more diverse training samples for those specific classes rather than a bigger model."

---

Confidence Thresholds and Decision Logic

The Syntiant® NDP does not operate on a binary "Yes" or "No" logic. Instead, for every window of sensor data it processes, the neural engine outputs a Confidence Score—a probability value (typically between 0 and 1) representing how closely the current input matches the trained target pattern. The Confidence Threshold (T) is the software-defined "gate" that determines when this probability is high enough to trigger a system action.

A. The Mechanics of the Threshold The threshold acts as a filter between the autonomous NDP and the rest of the hardware system.

• The Decision Rule: A detection is only officially registered if the output score (S) meets the condition: S ≥ T.

• The Hardware Trigger: Only when this condition is met does the NDP toggle its physical Interrupt Request (IRQ) pin to wake the host controller.

B. The Balancing Act: Sensitivity vs. Precision Choosing the value of T is the final step in product "tuning." It allows you to adjust the behavior of the device without retraining the entire model.

• Lowering the Threshold (T↓):

  • Effect: Increases Recall. The device becomes more "sensitive" and is more likely to catch faint or distant signals.
  • Trade-off: Increases the False Acceptance Rate (FAR). The device will trigger more often on similar-sounding background noise, potentially decreasing battery life.

• Raising the Threshold (T↑):

  • Effect: Increases Precision. The device only triggers when it is extremely "certain" of a match.
  • Trade-off: Increases the False Rejection Rate (FRR). The user may find the device unresponsive unless they are very loud or close to the sensor.

C. Advanced Decision Logic: Match Windows To prevent a single "lucky" noise spike from causing a false trigger, eFabric™ allows for Temporal Decision Logic through match windows. Instead of triggering on a single frame that crosses the threshold, you can require a sustained detection.

The "M-of-N" Logic: The system can be configured to trigger only if M frames out of the last N frames have exceeded the confidence threshold.

• Example: "Only trigger the alarm if 3 out of the last 5 windows (60%) indicate a glass-break signature."

💡Pro-Tip: The Hysteresis Strategy

"In high-stakes environments, use a Dual-Threshold approach. Set a high threshold (Thigh = 0.9) to trigger the initial wake-up, but once the system is active, use a lower 'keep-alive' threshold (Tlow = 0.6) to continue tracking the event. This prevents the system from 'flickering' on and off in noisy conditions."